Adapting Bias by Gradient

Appropriate bias is widely viewed as the key to e cient learning and generalization I present a new algorithm the Incremental Delta Bar Delta IDBD algorithm for the learning of appropri ate biases based on previous learning experience The IDBD algorithm is developed for the case of a simple linear learning system the LMS or delta rule with a separate learning rate parameter for each input The IDBD algorithm adjusts the learning rate parameters which are an important form of bias for this system Because bias in this approach is adapted based on previous learning experience the appropriate testbeds are drifting or non stationary learning tasks For particular tasks of this type I show that the IDBD algo rithm performs better than ordinary LMS and in fact nds the optimal learning rates The IDBD algorithm extends and improves over prior work by Jacobs and by me in that it is fully incremen tal and has only a single free parameter This paper also extends previous work by presenting a derivation of the IDBD algorithm as gradient descent in the space of learning rate parameters Finally I o er a novel interpretation of the IDBD algorithm as an incremental form of hold one out cross validation